{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.ERNIE Tiny 3.0 Introduction\n",
    "\n",
    "[ERNIE Tiny Models](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0) are lightweight models obtained from Wenxin large model ERNIE 3.0 using distillation technology. The model structure is consistent with ERNIE 2.0, and has a stronger Chinese effect than ERNIE 2.0.\n",
    "\n",
    "For a detailed explanation of related technologies, please refer to the article [_解析全球最大中文单体模型鹏城-百度·文心技术细节_](https://www.jiqizhixin.com/articles/2021-12-08-9)\n",
    "\n",
    "# 2.Model Effects\n",
    "\n",
    "ERNIE Tiny 3.0 open sources five models: **ERNIE 3.0 _Base_** 、**ERNIE 3.0 _Medium_** 、 **ERNIE 3.0 _Mini_** 、 **ERNIE 3.0 _Micro_** 、 **ERNIE 3.0 _Nano_** ：\n",
    "\n",
    "- [**ERNIE 3.0-_Base_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams) (_12-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Medium_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams) (_6-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Mini_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams) (_6-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Micro_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams) (_4-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Nano_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams) (_4-layer, 312-hidden, 12-heads_)\n",
    "\n",
    "\n",
    "Below is the **precision-latency graph** of the small Chinese models in PaddleNLP. The abscissa represents the latency (unit: ms) tested on CLUE IFLYTEK dataset (maximum sequence length is set to 128), and the ordinate is the average accuracy on 10 CLUE tasks (including text classification, text matching, natural language inference, Pronoun disambiguation, machine reading comprehension and other tasks), among which the metric of CMRC2018 is Exact Match (EM), and the metric of other tasks is Accuracy. The closer the model to the top left in the figure, the higher the level of accuracy and performance.The top left model in the figure has the highest level of accuracy and performance.\n",
    "\n",
    "The number of parameters of the model are marked under the model name in the figure. For the test environment, see [Performance Test](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95) in details.\n",
    "\n",
    "\n",
    "precision-latency graph under CPU (number of threads: 1 and 8), batch_size = 32:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852121-2798b5c9-d122-4ac0-b4c8-da46b89b5512.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852129-bbe58835-8eec-45d5-a4a9-cc2cf9a3db6a.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "precision-latency graph under CPU (number of threads: 1 and 8), batch_size = 1:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852106-658e18e7-705b-4f53-bad0-027281163ae3.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852112-4b89d675-7c95-4d75-84b6-db5a6ea95e2c.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "precision-latency graph under GPU, batch_size = 32, 1:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854679-3247f42e-8716-4a36-b5c6-9ce4661b36c7.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854670-57878b34-c213-47ac-b620-aaaec082f435.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "从图上可看出，ERNIE 3.0 系列轻量级模型在精度和性能上的综合表现已全面领先于 UER-py、Huawei-Noah 以及 HFL 的中文模型。且当 batch_size=1、预测精度为 FP16 时，在 GPU 上宽且浅的模型的推理性能更有优势。\n",
    "\n",
    "\n",
    "As can be seen from the figure, the comprehensive performance of the ERNIE Tiny 3.0 models has been comprehensively ahead of UER-py, Huawei-Noah and HFL in terms of accuracy and performance. And when batch_size=1 and the precision mode is FP16, the inference performance of the wide and shallow model on the GPU is more advantageous.\n",
    "\n",
    "在 CLUE **验证集**上评测指标如下表所示：\n",
    "The precision data on the CLUE **validation set** are shown in the following table:\n",
    "\n",
    "<table style=\"width:100%;\" cellpadding=\"2\" cellspacing=\"0\" border=\"1\" bordercolor=\"#000000\">\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center;vertical-align:middle\">\n",
    "                <span style=\"font-size:18px;\">Arch</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">Model</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AVG</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AFQMC</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">TNEWS</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">IFLYTEK</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">OCNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CLUEWSC2020</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CSL</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMRC2018</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CHID</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">C<sup>3</sup></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=3 align=center> 24L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Large-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.03</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>85.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>93.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.53</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.22/91.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.54</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Large-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.23</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.33</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95/90.31</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext-large</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.61</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">90.79</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.58/89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.26</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 20L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>ERNIE 3.0-Xbase-zh</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>78.39</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.16</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.55</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>61.87</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.40</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.82</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.99/93.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>86.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.98</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=9 align=center> 12L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams\">\n",
    "                        ERNIE 3.0-Base-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.10</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71/90.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>77.88</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.47</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.86</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.41</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>89.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.42</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.88/90.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.68</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.98</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE-Gram-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.87</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.82/90.38</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.69</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">Langboat/Mengzi-BERT-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.76</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.16</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.04/88.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.70</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.08/87.46</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.19</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.84</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.25</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.68</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.32/87.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.60</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.92</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.39/88.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.03</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">BERT-Base-Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.30/86.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.38</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.89</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.87/84.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.76</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 8L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Medium</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.10</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.63/78.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.84</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=5 align=center> 6L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams\">\n",
    "                        ERNIE 3.0-Medium-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.49</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>73.37</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>57.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.64</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.28</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>65.83/87.30</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>69.73</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HLF/RBT6, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.72/84.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.85</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">TinyBERT<sub>6</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.70</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.12</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.03/83.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.11</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoFormerV2 Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>60.72</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.37</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.97/83.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.66</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.41</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-L6-H768</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.54</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.74/75.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.40</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 6L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams\">\n",
    "                        ERNIE 3.0-Mini-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.24</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.30</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53/81.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.60</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "               <tr>\n",
    "            <td rowspan=1 align=center> 4L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT4, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.42</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.30/81.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.45</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.552</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.75/69.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">50.92</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams\">\n",
    "                    ERNIE 3.0-Micro-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">64.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.15</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.77/77.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=2 align=center> 4L312H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams\">\n",
    "                        ERNIE 3.0-Nano-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.51</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>54.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>48.36</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.61</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.93</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>52.00/76.35</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>55.11</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">TinyBERT<sub>4</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.94</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.04/69.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">52.18</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L256H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Mini</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">41.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">5.96/17.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBTL3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.50/80.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.03</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.56</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.73/78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.93</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 2L128H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Tiny</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">44.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">20.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">3.08/14.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">23.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">28.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "    <tbody>\n",
    "</table>\n",
    "<br />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "析全球最大中文单体模型鹏城-百度·文心技术细节_](https://www.jiqizhixin.com/articles/2021-12-08-9)\n",
    "\n",
    "# 2.Model Effects\n",
    "\n",
    "ERNIE Tiny 3.0 open sources five models: **ERNIE 3.0 _Base_** 、**ERNIE 3.0 _Medium_** 、 **ERNIE 3.0 _Mini_** 、 **ERNIE 3.0 _Micro_** 、 **ERNIE 3.0 _Nano_** ：\n",
    "\n",
    "- [**ERNIE 3.0-_Base_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams) (_12-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Medium_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams) (_6-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Mini_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams) (_6-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Micro_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams) (_4-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Nano_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams) (_4-layer, 312-hidden, 12-heads_)\n",
    "\n",
    "\n",
    "Below is the **precision-latency graph** of the small Chinese models in PaddleNLP. The abscissa represents the latency (unit: ms) tested on CLUE IFLYTEK dataset (maximum sequence length is set to 128), and the ordinate is the average accuracy on 10 CLUE tasks (including text classification, text matching, natural language inference, Pronoun disambiguation, machine reading comprehension and other tasks), among which the metric of CMRC2018 is Exact Match (EM), and the metric of other tasks is Accuracy. The closer the model to the top left in the figure, the higher the level of accuracy and performance.The top left model in the figure has the highest level of accuracy and performance.\n",
    "\n",
    "The number of parameters of the model are marked under the model name in the figure. For the test environment, see [Performance Test](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95) in details.\n",
    "\n",
    "\n",
    "precision-latency graph under CPU (number of threads: 1 and 8), batch_size = 32:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852121-2798b5c9-d122-4ac0-b4c8-da46b89b5512.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852129-bbe58835-8eec-45d5-a4a9-cc2cf9a3db6a.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "precision-latency graph under CPU (number of threads: 1 and 8), batch_size = 1:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852106-658e18e7-705b-4f53-bad0-027281163ae3.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852112-4b89d675-7c95-4d75-84b6-db5a6ea95e2c.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "precision-latency graph under GPU, batch_size = 32, 1:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854679-3247f42e-8716-4a36-b5c6-9ce4661b36c7.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854670-57878b34-c213-47ac-b620-aaaec082f435.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "从图上可看出，ERNIE 3.0 系列轻量级模型在精度和性能上的综合表现已全面领先于 UER-py、Huawei-Noah 以及 HFL 的中文模型。且当 batch_size=1、预测精度为 FP16 时，在 GPU 上宽且浅的模型的推理性能更有优势。\n",
    "\n",
    "\n",
    "As can be seen from the figure, the comprehensive performance of the ERNIE Tiny 3.0 models has been comprehensively ahead of UER-py, Huawei-Noah and HFL in terms of accuracy and performance. And when batch_size=1 and the precision mode is FP16, the inference performance of the wide and shallow model on the GPU is more advantageous.\n",
    "\n",
    "在 CLUE **验证集**上评测指标如下表所示：\n",
    "The precision data on the CLUE **validation set** are shown in the following table:\n",
    "\n",
    "<table style=\"width:100%;\" cellpadding=\"2\" cellspacing=\"0\" border=\"1\" bordercolor=\"#000000\">\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center;vertical-align:middle\">\n",
    "                <span style=\"font-size:18px;\">Arch</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">Model</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AVG</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AFQMC</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">TNEWS</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">IFLYTEK</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">OCNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CLUEWSC2020</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CSL</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMRC2018</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CHID</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">C<sup>3</sup></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=3 align=center> 24L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Large-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.03</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>85.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>93.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.53</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.22/91.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.54</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Large-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.23</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.33</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95/90.31</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext-large</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.61</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">90.79</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.58/89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.26</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 20L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>ERNIE 3.0-Xbase-zh</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>78.39</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.16</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.55</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>61.87</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.40</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.82</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.99/93.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>86.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.98</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=9 align=center> 12L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams\">\n",
    "                        ERNIE 3.0-Base-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.10</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71/90.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>77.88</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.47</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.86</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.41</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>89.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.42</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.88/90.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.68</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.98</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE-Gram-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.87</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.82/90.38</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.69</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">Langboat/Mengzi-BERT-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.76</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.16</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.04/88.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.70</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.08/87.46</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.19</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.84</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.25</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.68</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.32/87.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.60</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.92</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.39/88.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.03</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">BERT-Base-Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.30/86.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.38</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.89</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.87/84.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.76</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 8L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Medium</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.10</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.63/78.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.84</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=5 align=center> 6L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams\">\n",
    "                        ERNIE 3.0-Medium-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.49</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>73.37</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>57.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.64</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.28</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>65.83/87.30</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>69.73</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HLF/RBT6, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.72/84.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.85</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">TinyBERT<sub>6</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.70</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.12</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.03/83.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.11</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoFormerV2 Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>60.72</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.37</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.97/83.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.66</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.41</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-L6-H768</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.54</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.74/75.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.40</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 6L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams\">\n",
    "                        ERNIE 3.0-Mini-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.24</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.30</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53/81.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.60</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "               <tr>\n",
    "            <td rowspan=1 align=center> 4L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT4, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.42</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.30/81.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.45</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.552</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.75/69.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">50.92</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams\">\n",
    "                    ERNIE 3.0-Micro-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">64.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.15</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.77/77.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=2 align=center> 4L312H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams\">\n",
    "                        ERNIE 3.0-Nano-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.51</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>54.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>48.36</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.61</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.93</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>52.00/76.35</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>55.11</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">TinyBERT<sub>4</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.94</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.04/69.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">52.18</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L256H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Mini</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">41.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">5.96/17.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBTL3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.50/80.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.03</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.56</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.73/78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.93</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 2L128H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Tiny</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">44.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">20.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">3.08/14.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">23.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">28.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "    <tbody>\n",
    "</table>\n",
    "<br />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.How to Use the Model\n",
    "\n",
    "Install latest package of paddlenlp \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:50:18.329409Z",
     "iopub.status.busy": "2022-11-11T06:50:18.328968Z",
     "iopub.status.idle": "2022-11-11T06:53:07.814814Z",
     "shell.execute_reply": "2022-11-11T06:53:07.813775Z",
     "shell.execute_reply.started": "2022-11-11T06:50:18.329379Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install paddlenlp --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning\n",
    "\n",
    "The pre-trained models released by ERNIE 3.0 cannot be directly used for prediction, and the pre-trained models need to be fine-tuned using task-specific data.\n",
    "\n",
    "Using PaddleNLP, you only need one line code to get the ERNIE Tiny 3.0 models, and then you can fine-tune it under your own task data to obtain task-specific models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:53:07.817337Z",
     "iopub.status.busy": "2022-11-11T06:53:07.816843Z",
     "iopub.status.idle": "2022-11-11T06:53:56.003766Z",
     "shell.execute_reply": "2022-11-11T06:53:56.002942Z",
     "shell.execute_reply.started": "2022-11-11T06:53:07.817293Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from paddlenlp.transformers import *\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# For Text Classification Task\n",
    "seq_cls_model = AutoModelForSequenceClassification.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# For Token Classification Task\n",
    "token_cls_model = AutoModelForTokenClassification.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# For Question Answering Task\n",
    "qa_model = AutoModelForQuestionAnswering.from_pretrained(\"ernie-3.0-medium-zh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following script can start to fine-tune **ERNIE 3.0-Medium** on the CLUE IFLYTEK (text classification) dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:53:56.006231Z",
     "iopub.status.busy": "2022-11-11T06:53:56.005136Z",
     "iopub.status.idle": "2022-11-11T06:55:21.095442Z",
     "shell.execute_reply": "2022-11-11T06:55:21.094490Z",
     "shell.execute_reply.started": "2022-11-11T06:53:56.006195Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!git clone https://gitee.com/paddlepaddle/PaddleNLP.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:56:23.255848Z",
     "iopub.status.busy": "2022-11-11T06:56:23.255438Z",
     "iopub.status.idle": "2022-11-11T06:58:21.333380Z",
     "shell.execute_reply": "2022-11-11T06:58:21.332474Z",
     "shell.execute_reply.started": "2022-11-11T06:56:23.255822Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 分类任务\n",
    "# 该脚本共支持 CLUE 中 7 个分类任务，超参不全相同，因此分类任务中的超参配置利用 config.yml 配置\n",
    "!python PaddleNLP/model_zoo/ernie-3.0/run_seq_cls.py  \\\n",
    "    --task_name iflytek \\\n",
    "    --model_name_or_path ernie-3.0-medium-zh \\\n",
    "    --do_train\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Compression\n",
    "\n",
    "If there is a need to deploy the model online, you can further compress the model size. You can use the model compression plans and API to compress the fine-tuned models.\n",
    "\n",
    "The usage of the model compression API can be found in [documentation](../../docs/compression.md). Similarly, the model compression API also supports natural language processing tasks such as classification (including text classification, text matching, natural language inference, pronoun disambiguation, etc.), token classification, machine reading comprehension and information extraction and so on.\n",
    "\n",
    "The model exported after compression can be used directly for deployment.\n",
    "\n",
    "\n",
    "## 部署\n",
    "\n",
    "我们为 ERNIE 3.0 提供了[多种部署方案](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E9%83%A8%E7%BD%B2)，可以满足不同场景下的部署需求，请根据实际情况进行选择：\n",
    "\n",
    "\n",
    "We provide [various deployment scenarios](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E9%83%A8%E7%BD%B2) for ERNIE 3.0, which can meet the deployment requirements in different tasks, please choose according to the actual situation:\n",
    "\n",
    "<p align=\"center\">\n",
    "        <img width=\"700\" alt=\"image\" src=\"https://user-images.githubusercontent.com/26483581/175260618-610a160c-270c-469a-842c-96871243c4ed.png\">\n",
    "</p>\n",
    "\n",
    "# 4.Model Principles\n",
    "\n",
    "### Online Distillation Technology\n",
    "\n",
    "In the process of model learning, online distillation technology periodically transmits knowledge signals to several student models for simultaneous training, thereby producing student models of multiple sizes at one time in the distillation stage. Compared with the traditional distillation technology, this technology greatly saves the computing power consumption caused by the extra distillation calculation of the large model and the repeated knowledge transfer of multiple students.\n",
    "\n",
    "This novel distillation method takes advantage of the scale advantage of the Wenxin large model, and ensures the effect and size richness of the student model after the distillation is completed, which is convenient for application scenarios with different performance requirements. In addition, due to the huge gap between the model size of the Wenxin model and the student model, the model distillation is extremely difficult or even easy to fail. To this end, by introducing the technology of the teaching assistant model for distillation, the teaching assistant is used as a bridge for knowledge transfer to shorten the problem that the expression space between the student model and the large model is too large, thereby promoting the improvement of distillation efficiency.\n",
    "\n",
    "For more technical details, please refer to the paper:\n",
    "- [ERNIE-Tiny: A Progressive Distillation Framework for Pretrained Transformer Compression](https://arxiv.org/abs/2106.02241)\n",
    "- [ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://arxiv.org/abs/2112.12731)\n",
    "\n",
    "<p align=\"center\">\n",
    "        <img width=\"644\" alt=\"image\" src=\"https://user-images.githubusercontent.com/1371212/168516904-3fff73e0-010d-4bef-adc1-4d7c97a9c6ff.png\" title=\"ERNIE 3.0 Online Distillation\">\n",
    "</p>\n",
    "\n",
    "\n",
    "\n",
    "# 5.Related papers and citations\n",
    "\n",
    "\n",
    "```text\n",
    "@article{sun2021ernie,\n",
    "  title={Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation},\n",
    "  author={Sun, Yu and Wang, Shuohuan and Feng, Shikun and Ding, Siyu and Pang, Chao and Shang, Junyuan and Liu, Jiaxiang and Chen, Xuyi and Zhao, Yanbin and Lu, Yuxiang and others},\n",
    "  journal={arXiv preprint arXiv:2107.02137},\n",
    "  year={2021}\n",
    "}\n",
    "\n",
    "@article{su2021ernie,\n",
    "  title={Ernie-tiny: A progressive distillation framework for pretrained transformer compression},\n",
    "  author={Su, Weiyue and Chen, Xuyi and Feng, Shikun and Liu, Jiaxiang and Liu, Weixin and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng},\n",
    "  journal={arXiv preprint arXiv:2106.02241},\n",
    "  year={2021}\n",
    "}\n",
    "\n",
    "@article{wang2021ernie,\n",
    "  title={Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation},\n",
    "  author={Wang, Shuohuan and Sun, Yu and Xiang, Yang and Wu, Zhihua and Ding, Siyu and Gong, Weibao and Feng, Shikun and Shang, Junyuan and Zhao, Yanbin and Pang, Chao and others},\n",
    "  journal={arXiv preprint arXiv:2112.12731},\n",
    "  year={2021}\n",
    "}\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
